User interface system, scene description generating device and method, scene description converting device and method, recording medium, and sending medium

ABSTRACT

A user interface system comprises a server having a scene description converting device for converting scene description containing input user interaction into converted scene description while leaving parts containing the user interaction and a scene description decoding device for decoding converted scene description into decoded scene description, a remote terminal having a scene description decoding device for decoding scene description and converted scene description sent from the server and a display device for displaying decoded scenes and a user input device for receiving user input according to this display, and a display terminal for displaying decoded scenes sent from the server. Thus, decoding can be enabled at terminals having inferior decoding capabilities and display capabilities.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of co-pending U.S. applicationSer. No. 09/793,160, filed on Feb. 26, 2001, which claims priority toJapanese Application No. 2000-055053, filed Feb. 29, 2000, both of whichare incorporated herein by reference to the extent permitted by law.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a user interface which uses scenedescription information, a scene description information generatingdevice and method for generating scene description information, a scenedescription information converting device and method for convertingscene description information, and a sending medium for sending scenedescription information and a recording medium for recording scenedescription information.

2. Description of the Related Art

There are contents described with scene description enabling interactionby user input, such as digital TV broadcasting and DVD (DigitalVideo/Versatile Disk), Internet home pages described with HyperTextMarkup Language (hereafter referred to as “HTML”) or the like, BinaryFormat for the Scene (hereafter referred to as “MPEG-4 BIFS”) which is ascene description format stipulated in ISO/IEC14496-1, Virtual RealityModeling Language (hereafter referred to as “VRML”) which is stipulatedin ISO/IEC14472, and so forth. The data of such contents will hereafterbe referred to as “scene description”. Scene description also includesthe data of audio, images, computer graphics, etc., used within thecontents.

FIGS. 18 through 20 illustrate an example of scene description, takingVRML as an example.

FIG. 18 illustrates the contents of scene description. With VRML, scenedescription is text data such as shown in FIG. 18, and MPEG-4 BIFS scenedescription is the text data that has been encoded to binarized data.VRML and MPEG-4 BIFS scene description is represented by basicdescription units called nodes. In FIG. 18, nodes are underlined. Nodesare units for describing objects and the linkage relation of objects andthe like, containing data called fields for illustrating the propertiesand attributes of the nodes. For example, the Transform node 302 in FIG.18 is a node capable of specifying three-dimensional coordinatesconversion, and the amount of parallel movement of the point of originof the coordinates can be specified in the Translation filed 303. Thereare also fields capable of specifying other nodes, so the configurationof the scene description has a tree configuration such as shown in FIG.19. In FIG. 19, the ovals represent nodes, dotted lines between thenodes represent event propagation paths, and solid lines between thenodes represent the parent-child relations of the nodes. A node whichrepresents a field of the parent node thereof is called a child node.For example, in the Transform node 302 in FIG. 18, there is a Childrenfield 304 indicating a child node group which is subjected tocoordinates conversion by the Transform node, and the TouchSensor node305 and Shape node 306 are grouped as children nodes. A node which thusgroups child nodes to a Children filed is called a grouping node. In thecase of VRML, a grouping node is a node defined in ISO/IEC14772-1Section 4.6.5, and indicates a node having a field comprising a list ofnodes.

That is, a grouping node has a field containing a list of childrennodes. Each grouping node defines a positional space for children. Thispositional space relates to the positional space of the nodes whereingrouping nodes are children. Such a node is referred to as a parentnode. This means that conversion descends the scene chart hierarchy. Asdefined in ISO/IEC14772-1 Section 4.6.5. There are special exceptionswherein the field name is not Children, but in the followingdescription, the Children field will be understood to encompass suchexceptions as well.

In order to position an object to be displayed within a scene, the noderepresenting the object is grouped along with a node representingattributes, and further grouped with a node indicating positionallocation. The object which the Shape node 306 in FIG. 18 represents hasparallel movement specified by the Transform node 302 which is theparent node thereof applied thereto, and positioned in the scene. Thescene description in FIG. 18 contains a Sphere node 307 representing asphere, a Box node 312 representing a cube, a cone node 317 representinga cone, and a Cylinder node 322 representing a cylinder, with theresults of decoding and displaying the scene description being such asshown in FIG. 20.

The scene description may also contain user interaction. The ROUTE shownin FIG. 18 represents propagation of events. The ROUTE 323 indicatesthat in the event that the TouchTime field of the TouchSensor node 305to which an identifier called TOUCHS has been appropriated changes, thevalue thereof is propagated as an event to the StartTime field of theTimeSensor node 318 to which an identifier called TIMES has beenappropriated. In the event that the user has selected the Shape node 306which has been grouped with the Children field 304 of the Transform node302 which is the parent node of the TouchSensor node 305, theTouchSensor node 305 outputs the selected time as a TouchTime event. Asensor which is grouped with and works with a Shape node attached thusby a grouping node will be referred to as a Sensor node. A Sensor nodeis what ISO/IEC14772-1 Section 4.6.7.3 calls Pointing-device sensors,and attached Shape nodes are Shape nodes grouped with the parent node ofa Sensor node. That is, a Pointing-device sensor is for detecting apointing event wherein the user clicks on a shape such as a touchsensor, for example.

On the other hand, for one second from startTime, the TimeSensor node318 outputs the elapsed time as a fraction_changed event. Thefraction_changed event which represents the elapsed time output from theTimeSensor node 318 is propagated by the ROUTE 324 to the set_fractionfield of the ColorInterpolator node 319 to which an identifier calledCOL has been appropriated. The ColorInterpolator node 319 has functionsfor linear interpolation of RGB color-space values. The key and keyValuefields of the ColorInterpolator node 319 represent that in the eventthat the value of the set_fraction field which is input is 0, eventoutput of the RGB value [000] as value_changed is made, and that in theevent that the value of the set_fraction field which is input is 1,event output of the RGB value [111] as value_changed is made. In theevent that the value of the set_fraction field which is input is between0 and 1, event output of a value subjected to linear interpolation ofthe RGB value between [000] and [111], as value_changed, is made. Thatis to say, in the event that the value of the input set_fraction fieldwhich is input is 0.2, there is event output of the RGB value [0.2 0.20.2] as value_changed. The value value_changed as the results of linearinterpolation is propagated by the ROUTE 325 to the diffuseColor fieldof the Material node 314 to which has been appropriated an identifiercalled MAT. This diffuseColor represents the diffusion color of theobject surface which the Shape node 311 to which the Material node 314belongs represents. Event propagation by the above ROUTE 323, ROUTE 324,and ROUTE 325 realizes user interaction wherein the RGB values of adisplayed cube change from [000] to [111] for one second immediatelyfollowing the user selecting the displayed sphere.

FIG. 21 shows an example of a system for viewing and listening to scenedescriptions of contents described with a scene description methodenabling including interaction by user input, such as digital TVbroadcasting and DVD, Internet home pages described with HTML, MPEG-4BIFS, VRML, and so forth.

The server C01 takes the scene description C00 as input, and in theevent that the server itself comprises a decoding device for scenedescription, the scene description C00 is decoded and displayed on thedisplay terminal C13. Examples of the server C01 include a scenedescription re-distributing device or home server, digital TV broadcastsetup box, personal computer, and so forth. Normally, a user inputdevice C09 such as a mouse or keyboard is used to enable user input fora scene description containing user interaction. There are also caseswherein scene description is distributed to an external remote terminalC07. At this time, the remote terminal C07 may not necessarily havesufficient decoding capabilities and display capabilities for the scenedescription, and also there is the problem that sufficient sendingcapacity may not be secured for distribution.

The remote terminal C07 may have capabilities as a user input device. Insuch cases, the user input information C11 which has been input on theremote terminal C07 is transmitted to the server C01, reflected in thedecoding of the scene description at the server C01, and consequentlythe decoded scene C12 which reflects the user input is also displayed onthe display terminal C13.

FIG. 22 shows the configuration of a user interface system comprisingthe remote terminal having user input capabilities shown in FIG. 21.

In the event that the server D01 comprises a scene description decodingdevice D04, the scene description input D00 is decoded and the decodedscene D12 is displayed on the display terminal D13. On the other hand,the server D01 transmits the scene description D00 to the remoteterminal D07 via the transmitting/receiving device D06. The scenedescription D00 may be temporarily stored in the scene descriptionstoring device D05.

The remote terminal D07 receives the scene description D00 with thetransmitting/receiving device D06 b, decodes with the scene descriptiondecoding device D04 b, and displays with the display device D08. Thescene description D00 may be temporarily stored in the scene descriptionstoring device D05 b. In the event that the remote terminal D07 has userinput functions, the remote terminal D07 accepts user input D10 from theuser input device D09, and sends this as user input information D11representing user-selected position and the like to the scenedescription decoding device D04 b. The scene description decoding deviceD04 b decodes the scene description D00 based on the user inputinformation D11, thereby displaying decoded results reflecting the userinput on the display device D08. On the other hand, the remote terminalD07 may transmit the user input information D11 to the server D01 viathe transmitting/receiving device D06 b. In the event that the serverD01 comprises a scene description decoding device D04, the scenedescription decoding device D04 of the server D01 decodes the scenedescription D00 based on the user input information D11, therebydisplaying decoded scene D12 reflecting the user input D10 on thedisplay device D13.

With regard to viewing and listening to contents described with a scenedescription method enabling including interaction by user input, such asdigital TV and DVD, Internet home pages described with HTML and thelike, MPEG-4 BIFS, VRML, and so forth, there is demand for arrangementswherein decoding and display can be performed on terminals with inferiordecoding capabilities and display capabilities. At the time ofre-distribution with low-capacity media, there has also been the problemthat scene descriptions with large data could not be sent, or requiredsending media with large capacity.

Also, with regard to viewing and listening to contents comprising scenedescription containing user interaction, there has been the need for theuser to operate an input device such as a mouse toward the screen (inthe case of TV). Or, in the case of receiving with a PC, the user hashad to sit by the screen and operate a keyboard or mouse. Accordingly, auser interface system could be conceived wherein all contents aredisplayed on the remote terminal for the user to make input on theremote terminal, but a great deal of the contents usually are notdirectly related to user input, and transmitting all of this data to theuser terminal would necessitate great sending capacity for sending tothe remote terminal, and further require high decoding capabilities anddisplay capabilities for the remote terminal.

SUMMARY OF THE INVENTION

In order to solve the above problems, a first aspect of the presentinvention is a user interface system using scene description informationincluding user interaction, comprising: a server having converting meansfor converting scene description information; and a remote terminalhaving decoding means for decoding scene description information anddisplay means for displaying scenes; wherein the server sends scenedescription information converted by the converting means thereof to theremote terminal, and the remote terminal decodes with the decoding meansthereof and displays on the display means thereof, the converted scenedescription information.

According to a second aspect of the present invention, a scenedescription generating device for generating scene descriptioninformation including user interaction comprises encoding means forencoding the scene description information with an identifier wherebyjudgment can be made regarding portions to be left and portions to bedeleted at the time of converting the generated scene descriptioninformation later.

According to a third aspect of the present invention, a scenedescription generating device for generating scene descriptioninformation including user interaction comprises encoding means foroutputting beforehand other scene description information regardingportions to be left and portions to be deleted at the time of convertingthe scene description information later.

According to a fourth aspect of the present invention, a scenedescription generating method for generating scene descriptioninformation including user interaction comprises an encoding step forencoding the scene description information with an identifier wherebyjudgment can be made regarding portions to be left and portions to bedeleted at the time of converting the generated scene descriptioninformation later.

According to a fifth aspect of the present invention, a scenedescription generating method for generating scene descriptioninformation including user interaction comprises an encoding step foroutputting beforehand other scene description information regardingportions to be left and portions to be deleted at the time of convertingthe generated scene description information later.

According to a sixth aspect of the present invention, a scenedescription converting device for converting scene descriptioninformation including user interaction comprises converting meanswherein scene description information encoded including an identifierwhereby portions to be left and portions to be deleted can be judged isinput, and selection of the scene description information is made basedon the identifier, thereby performing conversion.

According to a seventh aspect of the present invention, a scenedescription converting device for converting scene descriptioninformation including user interaction comprises converting meanswherein scene description information regarding which portions to beleft and portions to be deleted have been separated is input, theconverting means performing converting by selecting the portions to beleft and portions to be deleted.

According to an eighth aspect of the present invention, a scenedescription converting method for converting scene descriptioninformation including user interaction comprises a converting stepwherein scene description information encoded including an identifierwhereby portions to be left and portions to be deleted can be judged isinput, and selection is made based on the identifier, thereby performingconversion.

According to a ninth aspect of the present invention, a scenedescription converting method for converting scene descriptioninformation including user interaction comprises a converting stepwherein scene description information regarding which portions to beleft and portions to be deleted have been separated is input, theconverting step performing converting by selecting the portions to beleft and portions to be deleted.

According to a tenth aspect of the present invention, a sending mediumsends scene description information including user interaction, whereinscene description information encoded during scene description with anidentifier whereby portions to be left and portions to be deleted can bejudged is sent.

According to an eleventh aspect of the present invention, a sendingmedium sends scene description information including user interaction,wherein scene description information wherein portions to be left andportions to be deleted have been separated beforehand is sent.

According to a twelfth aspect of the present invention, a recordingmedium sends scene description information including user interaction,wherein scene description information encoded during scene descriptionwith an identifier whereby portions to be left and portions to bedeleted can be judged is recorded.

According to a thirteenth aspect of the present invention, a recordingmedium sends scene description information including user interaction,wherein scene description information wherein portions to be left andportions to be deleted have been separated beforehand is recorded.

That is to say, according to the present invention, at the time ofviewing and listening to contents described with a scene descriptionmethod enabling including interaction by user input, such as digital TVand DVD, Internet home pages described with HTML and the like, MPEG-4BIFS, VRML, and so forth, conversion is performed for extracting onlypartial scene description relating to user interaction from the scenedescription input containing user interaction, thereby enabling decodingand display on terminals with inferior decoding capabilities and displaycapabilities.

Performing conversion which leaves parts relating to user interactionenables application as scene description to be transmitted to a remoteterminal having user input functions in a user interface system.

Also, according to the present invention, generating scene descriptionbeforehand by assuming that conversion of scene description according tothe present invention will be performed later allows the contents ofscene description following scene description conversion to becontrolled at the stage of generating the scene description.

Further, clearly specifying the parts to be extracted at the time ofconversion when generating the scene description not only facilitatesscene description conversion, but also enables arbitrary parts to beextracted regardless of interaction.

Also, separately distributing the scene description data for the remoteterminal extracted beforehand at the time of generating the scenedescription does away with the need for extracting processing at thereceiving side.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram representing a scene description audio-visualsystem to which a scene description converting device has been applied;

FIG. 2 is a flowchart illustrating scene description convertingprocedures for MPEG-4 BIFS and VRML;

FIG. 3 is a diagram illustrating the scene description contents beforeand after scene description conversion;

FIG. 4 is a diagram illustrating the scene description structure beforeand after scene description conversion;

FIG. 5 is a diagram illustrating decoding and display results ofconverted scene description;

FIG. 6 is a block diagram illustrating a re-distribution system of scenedescription corresponding to a second embodiment;

FIG. 7 is a block diagram representing a scene description audio-visualsystem corresponding to a third embodiment;

FIG. 8 is a block diagram representing a scene description generatingdevice corresponding to a fourth embodiment;

FIG. 9 is a diagram illustrating the scene description contents beforeand after scene description conversion corresponding to the fourthembodiment;

FIG. 10 is a diagram illustrating the scene description structure beforeand after scene description corresponding to the fourth embodiment;

FIG. 11 is a diagram illustrating the converted scene descriptiondecoding and display results corresponding to the fourth embodiment;

FIG. 12 is a diagram illustrating an example of scene descriptionaccording to a fifth embodiment;

FIG. 13 is a diagram illustrating scene description according to thefifth embodiment;

FIG. 14 is a diagram illustrating the scene description decoding anddisplay results corresponding to the fifth embodiment;

FIG. 15 is a block diagram illustrating an example of using 0Didstipulated in ISO/IEC14496-1 as a data identifier in the fifthembodiment;

FIG. 16 is a diagram illustrating the contents of scene descriptioncorresponding to a sixth embodiment;

FIG. 17 is a diagram illustrating the scene description data decodingand display results corresponding to the sixth embodiment;

FIG. 18 is a diagram illustrating the contents of scene description;

FIG. 19 is a diagram illustrating the structure of scene description;

FIG. 20 is a diagram illustrating scene description decoding and displayresults;

FIG. 21 is a diagram illustrating an example of a system at the time ofviewing and listening to conventional scene description; and

FIG. 22 is a block diagram illustrating the system configuration at thetime of viewing and listening to conventional scene descriptioncorresponding to FIG. 21.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

First, a user interface system using scene description informationcontaining user interaction will be described with reference to FIG. 1as a first embodiment of the present invention.

This user interface system is made up of a server 101, a remote terminal107 for displaying decoded scenes 112 b and also receiving user input110, and a display terminal 113 for displaying decoded scenes 112.

The server 101 has a scene description converting device 102 forconverting scene description 100, which is input scene descriptioninformation, into converted scene description 103, a scene descriptionstoring device 105 for storing input scene description 100 or convertedscene description 103 from the scene description converting device 102and sending this to the scene description converting device 102 or ascene description decoding device 104, and a scene description decodingdevice 104 for decoding scene description 100 or converted scenedescription 103 from the scene description converting device 102 or ascene description storing device 105 into converted scene description112, and sending this to a display terminal 113.

Also, the server 101 has a transmitting/receiving device 106 fortransmitting scene description 100 or converted scene description 103from the scene description storing device 105 or scene descriptionconverting device 102 to the remote terminal 107, and for receiving userinput information 111 from the remote terminal 107 and sending this tothe scene description decoding device 104.

The remote terminal 107 has a display device 108 for displaying thedecoded scenes 112 b, a user input device 109 of receiving user input110 according to this display, and a scene description decoding device104 b for decoding scene description 100 or converted scene description103 according to the user input information 111 from the user inputdevice 109.

Also, the remote terminal 107 has a scene description storing device 105b for storing and sending scene description 100 or converted scenedescription 103 to the scene description decoding device 104 b, and atransmitting/receiving device 106 b for receiving scene description 100or converted scene description 103 from the server 101 and sending thisto the scene description decoding device 104 b and scene descriptionstoring device 105, and for sending user input information 111 from theuser input device 109 to the server 101.

The display terminal 113 displays converted scenes 112 sent from theserver 101.

The first embodiment is the scene description converting device 102according to the present embodiment applied to the user interface systemat the time of viewing and listening to scene description, shown in FIG.22.

The scene description converting device 102 takes scene description 100as the input thereof and extracts parts in the scene description relatedto user interaction, thereby outputting converted scene description 103.The input and output scene description 100 and 103 of the scenedescription converting device 102 may be stored in the scene descriptionstoring device 105.

In the event that the server 101 has a scene description decoding device104, the scene description input 100 or converted scene description 103can be decoded and displayed with the display terminal 113.

The server 101 transfers the converted scene description 103 to theremote terminal 107 via the transmitting/receiving device 106. Theamount of data of the has been cut down by the scene descriptionconverting device 102 according to the present embodiment, so sendingcapacity lower than that of the conventional art suffices for thesending path from the server 101 to the remote terminal 107.

The remote terminal 107 decodes the converted scene description 103received from the server 101 with the scene description decoding device104 b and displays this with the display device 108. Parts unrelated touser interaction in scene description have been deleted beforehand bythe scene description converting device 102 according to the presentembodiment, so the scene description decoding device 104 b can decodescene description with decoding capabilities lower than what would benecessitated for decoding the scene description input 100 withoutchange. Further, the display device 108 is capable of display withdisplay capabilities lower than what would be necessitated fordisplaying the scene description input 100 without change.

In the event that the remote terminal 107 has a user input device 109,decoding and display of the scene description reflecting the user input110 is performed by sending the user input 110 as user input information111 regarding such as, for example, what position the user has selected,to the scene description decoding device 104 b. The scene descriptionconverting device 102 according to the present embodiment saves andconverts scene description relating to user interaction, and thus userinteraction the same as the scene description input 100 can be performedat the remote terminal 107.

At the time of reflecting the user input 110 on the remote terminal 107in the scene description decoding at the server 101 side as well, theremote terminal 107 transmits user input information 111 to the server101 via the transmitting/receiving device 106 b. The scene descriptiondecoding device 104 of the server 101 decodes the scene descriptionbased on the received user input information 111, thereby enablingconverted scenes 112 reflecting the user input 110 on the displayterminal 113. The scene description converting device 102 according tothe present embodiment saves and converts scene description relating touser interaction, and thus user input 110 made to the converted scenedescription 103 on the remote terminal 107 can be used without changefor decoding of the scene description input 100 on the server 101.

Incidentally, though not shown in the drawings, there are cases whereinthe user input information 111 is converted into information regardingwhich object in the scene description has been selected, by the scenedescription decoding device 104 b of the remote terminal 107, and thensent to the server.

The scene description converting device 102 according to the presentembodiment extracts parts within the scene description related to userinteraction, and deletes parts not related to user interaction. Theconversion procedures will be described with reference to FIG. 2, withan example of MPEG-4 BIFS and VRML.

First, in step S201, data representing propagation of events comprisinguser interaction in the scene description are not deleted but left. Inthe case of MPEG-4 BIFS or VRML, the nodes used for the ROUTErepresenting propagation of events are not deleted but left. ROUTE isnot deleted.

With scene description comprising a tree structure as with MPEG-4 BIFSor VRML, an effective tree structure must be saved even after datarepresenting propagation of events comprising user interaction has beenextracted, so subsequently, judgment is made from the top of the treestructure of the scene description toward the end thereof, regardingwhether to leave or delete nodes.

A parent node of a node which has been determined to be left in stepS202 is always left.

In step S203, in the event that the node regarding which judgment is tobe made is a node representing a Children field of a grouping node, theflow proceeds to step S204. Otherwise, the flow proceeds to step S206.

In step S204, in the event that the node regarding which judgment is tobe made is a Children field of a grouping node, judgment is made in stepS205 that this can be deleted in the event that it is not a Shape nodeattached to a Sensor node which has been judged to be let due to beingused by ROUTE in step S201.

For example, the TouchSensor node is a Sensor for detecting whether ornot the user has selected a Shape node grouped with a parent node, so inorder to save the functions of the TouchSensor, it is necessary to savethe attached Shape node.

In step S206, in the event that the node regarding which judgment is tobe made is not a Children field of a grouping node, judgment is made instep S207 that this can be deleted in the event that the parent node isto be deleted.

In step S208, in the event that all nodes have been subjected tojudgment of whether to delete or leave the nodes, the flow ends. In theevent that there are nodes not yet judges, the flow proceeds to stepS209.

In step S209, the object of judgment is shifted to the nodes not yetjudges, and the flow is repeated from step S202 again.

Incidentally, in steps S205 and 207, nodes regarding which judgment hasbeen made that the nodes can be deleted, do not necessarily have to bedeleted. For example, in the event that sufficient deletion has beenmade with regard to the sending capacity of the sending medium used forsending the converted scene description, and for the capabilities of thescene description decoding device and display device, no further nodesneed to be deleted.

FIGS. 3 through 5 show an example of converting the scene descriptionwith the processing procedures shown in FIG. 2. The scene descriptionbefore and after conversion shown in FIG. 3 is VRML, and the scenedescription is subjected to encoding into binary data with MPEG-4 BIFS.For the sake of description, an example of converting the same scenedescription as that shown in FIG. 18 is given here. The parts to bedeleted by conversion are struck through with double lines.

FIG. 4 illustrates the scene description structure before and afterconversion. First, due to the procedure step S201 shown in FIG. 2,decision has been made to leave the TouchSensor node 305 used withROUTE, the TimeSensor node 318, the ColorInterpolator node 319, and theMaterial node 314. Next, due to the procedure step S202, decision hasbeen made to leave the Transform node 302, Group node 300, Appearancenode 313, Shape node 311, and Transform node 310, which are parent nodesof the above nodes regarding which leaving has been decided.Subsequently, decision is made from the top of the tree structure of thescene description toward the end thereof, regarding whether to leave ordelete nodes. The unjudged Shape node 306 is a Children field of theTransform node 302 which is a grouping node and also the TouchSensornode is attached, so decision is made not to delete. The Sphere node 307and Appearance node 308 which are children nodes of the Shape node 306,and further the Material node 309 which is a grandchild node thereof, donot have their parent nodes deleted in step S206, and according decisionis made not to delete these nodes.

The Box node 312 is not a child of a grouping node and the Shape node311 which is the parent node thereof has been decided to not be deleted,so the decision is made not to delete the Box node 312. The Transformnode 315 is a child field of the Transform node 310 which is a groupingnode, but there is no Sensor attached, and accordingly judgment is madein step S205 that this can be deleted. The shape node 316 which is achild node thereof and the further child node Cone node 317 are judgedin step S207 that deletion may be made. In the same manner, makingjudgement with the procedures shown in FIG. 2 decides that the Transformnode 320, Shape node 321, and Cylinder node 322 can be deleted.

Consequently, the decoding and display results of the converted scenedescription are, as shown in FIG. 5, only a sphere and cube.

According to the converting method of the present embodiment, the partsrelating to user interaction in the scene description are saves, so theuser interaction wherein the color of the cube 311 consecutively changesfrom RGB=[000] to RGB=[111] for one second after the user selecting thesphere 306 is saved even after conversion.

The scene description converting device according to the presentembodiment is equally valid regarding the function of leaving partsrelating to user interaction for scene description input other than VRMLand MPEG-4 BIFS as well, and is by no means restricted to VRML andMPEG-4 BIFS.

A user interface system will be described as a second embodiment of thepresent invention, with reference to FIG. 6.

This user interface system comprises a server 401, and a remote terminal407 for displaying decoded scenes 412 b and also receiving user input410.

The server 401 comprises a scene description converting device 402 forconverting scene description 400 which is input scene descriptioninformation into converted scene description 403, a scene descriptionstoring device 405 for storing input scene description 400 and convertedscene description 403 from the scene description converting device 402and sending it to the scene description converting device 402, and atransmitting device 406 for transmitting the scene description 400 andconverted scene description 403 from the scene description convertingdevice 402 and scene description storing device 405 to the remoteterminal 407.

The remote terminal 407 has a display device 408 for displaying decodedscenes 412 b, a user input device 409 for receiving user input 410according to this display, and a scene description decoding device 404 bfor decoding scene description 400 and converted scene description 403according to user input information 411 from the user input device 409.

Also, the remote terminal 407 has a scene description storing device 405b for storing and sending scene description 400 or converted scenedescription 403 to the scene description decoding device 404 b, and areceiving device 406 b for receiving scene description 400 or convertedscene description 403 from the server 401 and sending this to the scenedescription decoding device 404 b and scene description storing device405 b.

The difference with the first embodiment is that the server 401 itselfdoes not comprise a scene description decoding device; rather, the scenedescription re-distributing system is arranged such that the convertedscene description converted by the scene description converting device402 us re-distributed to the remote terminal 407. The scene descriptionconverting device 402 is the same as that in the first embodiment.

As shown in FIG. 6, converting the scene description 400 with there-distribution server 401 by the scene description converting device402 according to the present embodiment and then re-distributing itallows the sending capacity at the time of re-distribution to besuppressed to a low level, and further enables decoding and displayingof scene description with the remote terminal 407 having inferiordecoding capabilities and display capabilities.

A third embodiment of the present invention will be described withreference to FIG. 7.

In the third embodiment, the decoding terminal 501 has a convertingdecoding device 502 which converts and decodes input scene description500 and sends the converted decoded scenes 512 to the display terminal513, a scene description storing device 505 for storing input scenedescription 500 and sending this to the scene description convertingdecoding device 502, and a user input device 509 for receiving userinput 510 and sending user input information 511 to the scenedescription converting decoding device 502.

The display device 513 displays the converted decoded scenes 512 sentfrom the decoding terminal 501.

The difference with the first and second embodiments is that instead ofconverting the scene description 500 and outputting this as convertedscene description, the scene description converting decoding device 502within the decoding terminal 501 containing the scene descriptionconverting functions performs both converting and decoding. That is tosay, parts containing interaction are extracted in the same manner aswith the description made regarding the first embodiment, and decodingis also performed at the same time. The method for extracting the partscontaining interaction is the same as that in the first embodiment. Thescene description converting decoding device 502 enables displaying theconverted decoded scenes 512 consisting only of parts of the interactionof the scene description being input on the display terminal 513.

A fourth embodiment of the present invention will be described withreference to FIG. 8.

In the fourth embodiment, a scene description generating device 615 hasa scene description encoding device 616 which encodes input scenario 614to form scene description 600, a scene description storing device 617for storing the scene description 600 from the scene descriptionencoding device 616, and a scene description data distributing device618 for distributing the scene description 600 sent from the scenedescription encoding device 616 and scene description storing device 617via a recording medium 619 or sending medium 620.

The server re-distributing server, and decoding terminal 601 receive thescene description 600 distributed via the recording medium 619 orsending medium 620.

In the third embodiment, the decoding terminal 501 has a scenedescription converting decoding device 502 which converts and decodesinput scene description 500 and sends the converted decoded scenes 512to the display terminal 513, a scene description storing device 505 forstoring input scene description 500 and sending this to the scenedescription converting decoding device 502, and a user input device 509for receiving user input 510 and sending user input information 511 tothe scene description converting decoding device 502.

The display device 513 displays the converted decoded scenes 512 sentfrom the decoding terminal 501.

The fourth embodiment relates to the scene description generating device615 which generates scene description. The scene description generatingdevice 615 has a scene description encoding device 616, takes scenario614 as input regarding what sort of scene is to be described, andgenerates the scene description 600. The scene description 600 may beeither text data or binary data, depending on the case. The scenedescription 600 may also be stored in the scene description storingdevice 617. The generated scene description 600 is input to the server101 shown in FIG. 1, the re-distributing server 401 shown in FIG. 6, andthe decoding terminal 501 shown in FIG. 7, via the recording medium 619or sending medium 620.

The scene description encoding device 616 according to the presentembodiment determines beforehand the parts to be deleted and the partsto be left so as to be distinguishable, for the time of converting scenedescription at the server which receives scene description, there-distribution server, and the decoding terminal.

FIGS. 9 through 11 describe scene description output by the scenedescription encoding device according to the present invention, takingVRML as an example. For the sake of description, the scene descriptionis the same between FIG. 3 and FIG. 18. The scene description encodingdevice according to the fourth embodiment attaches identifiersbeforehand to parts to be deleted at the scene description convertingdevice or scene description converting decoding device. FIG. 9 showsthat the Transform node 315 to which an identifier REM1 has beenattached, the Transform node 320 to which an identifier REM2 has beenattached, and all children nodes thereof, are to be deleted. Either thescene description converting device outputs the identifiers attached tothe nodes to be deleted along with the scene description, or identifierspredetermined with the scene description converting device or scenedescription converting decoding device beforehand is used, therebyenabling the scene description converting device or scene descriptionconverting decoding device to judge which parts of the scene descriptioninput to leave, and which parts to delete. Due to the scene descriptiongenerating device according to the present embodiment, the convertingprocessing of the scene description at the scene description convertingdevice or scene description converting decoding device does not requireprocessing such as described in FIG. 2, meaning that conversion can beperformed easily. Further, there is the advantage that the scenedescription following conversion can be determined at the time ofgenerating the scene description, regardless of whether interaction iscontained or not. For example, in the scene description shown in FIG. 9,not attaching the identifier REM2 to the Transform node 320 would allowthe cone of the Shape node 321 which is unrelated to interaction to bedisplayed as the converting and decoding results of the scenedescription converting device or scene description converting decodingdevice.

FIG. 10 illustrates the structure of the scene description before andfollowing conversion. The Transform node 320 to which the identifierREM1 is attached, and all the children nodes thereof, are deleted.

FIG. 11 shows the decoding and displaying results of the decoded scenedescription. As with the case of FIG. 5, the user interaction whereinthe color of the cube 311 consecutively changes from RGB=[000] toRGB=[111] for one second after the user selecting the sphere 306 issaved even after conversion.

The present embodiment is not dependent on the type of scene descriptionmethod. Rather, the present embodiment is effective in all scenedescription methods wherein identifiers capable of differentiating partsto be deleted or parts to be left can be embedded in the scenedescription. For example, with MPEG-4 BIFS, the same advantages can behad by using a nodeID stipulated in ISO/IEC14496-1 as the aboveidentifier.

The fifth embodiment of the present invention will be described withreference to FIGS. 12 through 14.

The fifth embodiment relates to a scene description generating devicefor generating scene description, and the configuration of the scenedescription generating device is the same as that shown in FIG. 8 withthe fourth embodiment. However, the scene description encoding deviceaccording to the fifth embodiment outputs parts to be left by conversionand parts to be deleted, as separate data beforehand, instead ofembedding identifiers for scene description conversion in the scenedescription.

FIGS. 12 through 14 describe scene description output by the scenedescription encoding device according to the fifth embodiment, takingVRML as an example. For the sake of description, the scene descriptioncontents are the same between FIG. 3, FIG. 9, and FIG. 18. The scenedescription encoding device according to the fifth embodiment outputsscene description with the data to be left and data to be deleted by thescene description converting device or scene description convertingdecoding device as data separated beforehand, as shown in FIGS. 12 and13A-13B. The VRML inline node makes reference to the scene descriptionof the external data using the URL (Uniform Resource Locator), and takesit in. The scene description shown in FIG. 12 makes reference to thescene description data1 shown in FIG. 13A by the inline node 315 b, andthe scene description data2 shown in FIG. 13B by the inline node 320 b,respectively. The scene description converting device or scenedescription converting decoding device taking as input the three sets ofscene description data shown in FIGS. 12 and 13A-13B is capable of usingall three sets of scene description to obtain the scene descriptiondecoding results shown in FIG. 14A, or use only the scene descriptionshown in FIG. 12 to obtain the scene description decoding results shownin FIG. 14B. Further, only the scene description shown in FIG. 13A maybe deleted to delete only the cone 316 from the scene descriptiondecoding results shown in FIG. 14A. Note that the present embodiment isequally effective in deleting either the part which makes reference tothe inline node or the part which is referred to.

The scene description converting device outputting the data identifiersfor identifying scene description data to be deleted or to be left,along with the scene description, or using data identifierspredetermined with the scene description converting device or scenedescription converting decoding device beforehand, enables the scenedescription converting device or scene description converting decodingdevice to judge which scene description of multiple input scenedescriptions to leave, and which scene description to ignore. The dataidentifiers may be either file names of scene description data, or inthe event of forming packets of the data may be any identifier wherebypackets can be identified.

An example of using 0Did, which is the identifier for ObjectDescriptorstipulated in ISO/IEC14496-1, as the above data identifier is shown inFIG. 15. The inline node in the scene description (BIFS) can specify thescene description (BIFS) data to be read in with the unique identifier0Did. 0Did is an ID appropriated to each descriptor calledObjectDescriptor, and information relating the object to which referenceis made is contained in the ObjectDescriptor data, of which scenedescription (BIFS) data to use. Accordingly, 0Did can be used as a dataidentifier with the present embodiment.

The scene description (BIFS) corresponding to the ObjectDescriptor dataof 0Did=1 shown by A in FIG. 15 is MPEG-4 BIFS data equivalent to FIG.12. The scene description (BIFS) corresponding to the ObjectDescriptordata of 0Did=2 shown by B in FIG. 15 is MPEG-4 BIFS data equivalent toFIG. 13A. The scene description (BIFS) corresponding to theObjectDescriptor data of 0Did=3 shown by C in FIG. 15 is MPEG-4 BIFSdata equivalent to FIG. 13B.

With the example shown in FIG. 15, the scene description (BIFS) data of0Did=2 and 0Did=3 can be deleted at the time of converting the scenedescription, either by the scene description generating deviceoutputting information to delete the scene description (BIFS) data of0Did=2 and 0Did=3, or by predetermining with the scene descriptionconverting device or scene description converting decoding devicebeforehand.

Also, even without using data identifiers, outputting divided scenedescription as other data beforehand using the scene descriptiongenerating device according to the present embodiment is effective infacilitating conversion processing by the scene description convertingdevice or scene description converting decoding device selecting andoutputting appropriate scene description data dependent on the sendingcapacity usable for re-distribution of the converted scene descriptionor capabilities of the decoding and display devices.

According to the scene description generating device of the presentembodiment, conversion processing by the scene description convertingdevice or scene description converting decoding device does not needprocessing such as shown in FIG. 2; rather, simply ignoring the scenedescription data to be deleted allows conversion to be performed easily.

This is further advantageous in that scene description followingconversion can be determined at the time of generating the scenedescription, regardless of whether or not interaction is containedtherein.

Further, dividing the scene description data into multiple sets allowsthe scene description which is the output of the scene description datadistributing device 618 to be recorded on an arbitrary recording medium,sent via an arbitrary sending medium, and the two to be arbitrarilymixed, as shown in FIG. 8. For example, an arrangement may be madewherein one set of scene description data is distributed by a sendingmedium for TV broadcasting, another set of scene description data isrecorded to a recording medium such as DVD, and both are used in acombined manner at a server, re-distributing server, or a decodingterminal.

The sixth embodiment of the present invention will be described withreference to FIGS. 16 and 17A-17B.

The sixth embodiment relates to the scene description generating devicefor generating scene description, with the configuration of the scenedescription generating device being the same as that shown in FIG. 8with the fourth embodiment. However, the scene description encodingdevice according to the sixth embodiment does away with the need forconversion processing by outputting only the parts to be left byconversion as other data beforehand.

FIGS. 16 and 17A-17B describe scene description output by the scenedescription encoding device according to the sixth embodiment, takingVRML as an example.

The entire FIG. 16 is a scene description contained in the scenedescription data D1, and the scene description data D2 in FIG. 16represents a scene description part contained in the scene descriptiondata D1. The scene description encoding device according to the sixthembodiment outputs both the scene description data D1 and scenedescription data D2. Decoding and displaying only the scene descriptiondata D1 yields that shown in FIG. 17A, and decoding and displaying onlythe scene description data D2 yields the shown in FIG. 17B with the coneremoved. The scene description of the scene description data D1 and thescene description of the scene description data D2 can each beindependently decoded and displayed, so all that is necessary for thescene description converting device or scene description convertingdecoding device is to select one of the scene descriptions.

The scene description encoding device outputting the data identifiersfor identifying the scene description data to be selected by the scenedescription converting device or scene description converting decodingdevice along with the scene description, or using identifierspredetermined with the scene description converting device or scenedescription converting decoding device beforehand, enables the scenedescription converting device or scene description converting decodingdevice to judge which of the multiple scene descriptions to use alone.The data identifiers are of the same format as that described with thefifth embodiment.

With the scene description encoding device according to the sixthembodiment, there is absolutely no need for the scene descriptionconverting device or scene description converting decoding device toperform any conversion processing relating to the contents of scenedescription. Accordingly, the output of the scene description convertingdevice or scene description converting decoding device can be decided atthe stage of generating scene description, regardless of whether or notthere is interaction, and optimal scene description can be createdaccording to the sending capacity for sending the converted scenedescription and the decoding and display capabilities.

Further, as with the fifth embodiment, dividing the scene descriptiondata into multiple sets allows the scene description which is the outputof the scene description data distributing device 618 to be recorded onan arbitrary recording medium, sent via an arbitrary sending medium, andthe two to be arbitrarily mixed, as shown in FIG. 8.

Thus, the present embodiment relates to generating of scenes comprisingstill image signals, motion image signals, audio signals, text, data,graphics data, etc., and containing interaction wherein input from usersis received, and to user input thereof, and also to equipment control.This art is suitably applied to, for example, performing user input atthe receiving side, interacting with scenes, controlling equipment,etc., at the time of playing from recording media such asmagneto-optical disks, magnetic disks, or magnetic tape, and displayingon a display or receiving contents of the Internet and broadcasts.

The present embodiment enables decoding and display to be made withterminals having inferior decoding capabilities or display capabilities,by performing conversion wherein only partial scene description relatingto user interaction is extracted, at the time of viewing and listeningto contents made up of scenes containing interaction from user input,such as digital TV broadcasts and DVD, HTML, MPEG-4 BIFS, VRML, and soforth. Also, scene descriptions are generated beforehand assuming thatscene description conversion will be performed later, so the contents ofscene description following scene description conversion can becontrolled at the stage of generating the scene descriptions, thusfacilitating conversion.

Note that all of the embodiments can be realized by hardware or bysoftware.

As described above, there have been demands for decoding and displayingwith terminals having inferior decoding capabilities and displaycapabilities regarding viewing and listening to contents described withscene description enabling interaction by user input to be containedtherein. Also, there have been problems in that scene descriptions withlarge amounts of data cannot be sent, or sending media with greatcapacity is needed, for re-distributing using sending media with smallcapacity.

According to the present invention, conversion is performed wherein onlypartial scene descriptions relating to user interaction are extractedfrom the scene description input containing user interaction, therebyenabling decoding and displaying with terminals having inferior decodingcapabilities and display capabilities. Alleviating the amount of data ofthe scene description is advantageous in that the sending capacitynecessary for sending and re-distributing the scene description can besuppressed to a low level.

Performing conversion which leaves parts relating to user interactionenables providing of a user interface system capable of maintaining userinteraction the same as before conversion with remote terminals havinguser input functions, as well.

Also, with the present invention, generating scene description such asto facilitate identifying parts to be deleted and parts to be leftbeforehand, by assuming that conversion of scene description accordingto the present invention will be performed later, allows the contents ofscene description following scene description conversion to becontrolled at the stage of generating the scene description. This isadvantageous in that the scene description conversion processing issimplified.

Further, outputting beforehand, at the time of generating, the parts tobe extracted at the time of conversion as other scene description data,or outputting both scene description before conversion and afterconversion, facilitates easy conversion since all that is necessary atthe time of conversion is to select the scene description data. This isalso advantageous in that screen description following conversion can bedecided at the time of generating the screen description, regardless ofwhether interaction is contained therein or not. Also, dividing thescene description data into multiple sets allows the scene descriptionwhich is the output of the scene description generating device to berecorded on an arbitrary recording medium, sent via an arbitrary sendingmedium, and be arbitrarily mixed.

1. A user interface system using scene description information includinguser interaction, said system comprising: a server having convertingmeans for converting scene description information; and a remoteterminal having decoding means for decoding scene descriptioninformation and display means for displaying scenes; wherein said serversends scene description information converted by the converting meansthereof to said remote terminal, and said remote terminal decodes withthe decoding means thereof and displays on the display means thereof,the converted scene description information, and wherein said convertingmeans leaves portions describing user interaction regarding said scenedescription information, and converts other portions so as to bedeleted; wherein said remote terminal has input means to which userinput information is input; and wherein said remote terminal sends tosaid server user input information that is input to said input meansaccording to a scene displayed on the display means thereof; and whereinsaid server performs conversion at the converting means thereof based onthe user input information.
 2. A user interface system according toclaim 1, wherein said portions describing user interaction are portionsnecessary for generation, propagation, reception, or execution ofevents.